Accelerating Deformable Convolution Networks with Dynamic and Irregular Memory Accesses

Deformable convolution networks (DCNs) proposed to address the image recognition with geometric or photometric variations typically involve deformable that convolves on arbitrary locations of input features. The change different inputs and induce considerable dynamic irregular memory accesses which cannot be handled by classic neural network accelerators (NNAs). Moreover, bilinear interpolation (BLI) operation is required obtain deformed features in DCNs also deployed existing NNAs directly. Although a general purposed processor (GPP) seated along can process convolution, processing GPP extremely slow due limited parallel computing capability massive additional data movement. To problem, we develop DCN accelerator support both standard convolution. Specifically, for DCNs, have output divided into tiles build tile dependency table (TDT) track at runtime. With TDT, further an on-chip scheduler handle efficiently. In addition, propose novel mapping strategy enable BLI apply layer fusion techniques more energy-efficient processing. According our experiments, achieves orders magnitude higher performance energy efficiency compared typical architectures including ARM, ARM+TPU, GPU 6.6% chip area penalty NNA.